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About the research 

Tracking our success: How TAPE institutes evaluate their effectiveness and efficient 
Josie Misko and Sian Halliday-Wynes 

This paper investigates how technical and further education (TAPE) institutes evaluate their 
effectiveness and efficiency. This issue is pertinent because TAPE institutes are increasingly 
expected to operate in an environment of high public accountability, greater fiscal restraint and 
expanded competition for government funding. Council of Australian Governments (COAG) 
suggestions for the implementation of ongoing market reform in the vocational education and 
training (VET) system will require institutes to further focus on effectiveness and efficiency. 

The authors interviewed nine TAPE institute directors and 59 of their senior and middle managers 
in South Australia, Queensland, New South Wales and Victoria. A support document of institute- 
specific case studies accompanies this paper. See <www.ncver.edu.au/publications/2169.html>. 

Key messages 

T Externally imposed requirements for funding accountability and regulatory compliance are the 
main drivers of institute processes and systems for monitoring and evaluating effectiveness and 
efficiency. This applies both to institutes within centralised governance arrangements and to 
those in devolved governance systems. 

T Regardless of governance structures, strong managerial leadership is critical to the evaluation of 
effectiveness and efficiency. 

T TAPE institutes engage with employers and industry bodies to better understand training 
demand, but employers do not always speak with one voice. 

T TAPE institutes have a general understanding of their markets and business. This may help 
them to survive in a more competitive environment, but they will require more robust systems 
for working out the cost-efficiency of their operations. 



Tom Karmel 



Managing Director 
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How TAPE institutes evaluate their 
effectiveness and efficiency 



Introduction 

Over the last two decades technical and further education (TAPE) institutes have been required to 
operate in an environment that has become less predictable. They have had to compete with other 
providers for government training funds (especially for apprenticeships and traineeships) as well as 
for commercial fee-for-service activities. There are reforms mooted by the Council of Australian 
Governments (COAG) to intensify competition for government funds and to simplify national 
regulatory frameworks for quality assurance outcomes. It is also proposed to give students more 
freedom to choose where they will get their training. The COAG reforms will require TAPE 
institutes to focus their attention on effectiveness and efficiency and to supplement government 
funds through commercial activity. TAPE directors and their management teams are anticipating 
the COAG reforms and realise that their task will be challenging and complex. 

In this study we examine the types of strategies and information sources TAPE institutes use to 
help them understand their performance and identify areas where improvement is needed. We 
examine effectiveness and efficiency from four angles: getting the plans right; understanding 
government and commercial business; meeting client needs; and meeting quality assurance 
requirements. 

We find that institute arrangements for monitoring their effectiveness and efficiency are mainly 
driven by external requirements for meeting negotiated targets imposed by government funding 
bodies and for meeting regulatory federal, state (and sometimes international) compliance 
frameworks. Professional accreditation and quality assurance also prompt attention to effectiveness 
and efficiency processes in institutes. These requirements necessitate the establishment of regular 
monitoring and reporting processes that help them to understand their business as well as the 
training environments in which they operate. 

Our findings indicate that institutes are embracing the concepts of evidence-based decision-making. 
This helps them to monitor their performance against government financial and non-financial 
targets, the volume and quality of training activity and outcomes, and the adequacy of their support 
and administration services. All are focused on remaining compliant with the Australian Quality 
Training Pramework 2007 (AQTP 2007)' and other formal registration and course accreditation 
requirements. 

The majority of institutes consulted are focused on cost-efficiency and effectiveness, with a number 
implementing formal processes for calculating the commercial viability of programs, based on in- 
built margins for attrition and profit. Some TAPE programs and services, while not always 
commercially viable, are seen as essential (including child care, student counselling and support). 



^ AQTF 2007 uses an outcomes-focused auditing approach. Institutes will be audited against indicators of performance (including 
student satisfaction, employer satisfaction and completions). 
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We have found that the more autonomous institutes (including the TAPE institutes that are 
statutory authorities in Queensland and stand-alone and dual-sector TAPE institutes in Victoria) do 
not act much differendy from institutes that belong to centralised systems. Both types of 
institutions are driven by the contracts they have with state training bodies and the need to meet 
AQTP 2007 requirements . 2 Similarities in their processes can be explained by the considerable 
information-sharing that exists between institutes within and across jurisdictions (including at 
service unit level). We cannot, however, disregard the considerable influence that is exerted on these 
processes by directors and their management teams. Having committed and energetic leadership at 
executive and managerial levels (including teaching and non-teaching functions) is a key factor in 
the establishment and maintenance of good processes for tracking progress against targets and for 
achieving objectives. 

TAPE institutes have been preparing themselves for competition ever since the training reform 
agenda of the early 1990s. They are relatively well placed to meet these challenges and have taken 
steps to ensure the commercial viability of their operations. What remains is to ensure that TAPE 
institutes improve access to, and use of, good information. 

Traditionally, TAPE has been expected and funded to provide affordable and accessible vocational 
education in metropolitan and country Australia. If we believe that this should be an enduring role 
for public providers, then we must recognise the critical challenges they face in a more competitive 
environment. TAPE institutes will need to ensure that they do not compromise the quality and 
scope of their provision (including responsibility for meeting needs of equity groups) by reducing 
costs too far in their efforts to win government tenders. 

We begin our discussion with a brief description of the way the information was collected. We then 
detail the various governance arrangements that apply to TAPE institutes; we examine how 
institutes go about getting their plans right and understanding their government and commercial 
business. We review arrangements for how TAPE institutes know they are meeting industry and 
student needs and how they ensure the quality of training and service provision. In aU cases we 
discuss areas for improvement. We end by drawing some conclusions. 



Our approach 

We base our findings on in-depth interviews with nine TAPE institute directors and 59 of their 
senior and middle managers.’ We have taken into account state and institute strategic plans, annual 
reports and documented processes. Case studies for each of the institutes in the study appear in the 
support document to this report. 

Our goal was to obtain insdtute-wide perspectives on strategic planning, financial review, 
marketing, quality assurance, student and staff support, administration and human resources. 
Educational managers (of trade and non-trade areas) were included to ensure that specific 
information about practical delivery issues was collected. 

The discussion is limited to the processes that institutes implement to achieve effectiveness and 
efficiency.’ This means that, although we can comment on the usefulness of processes and on gaps 
that remain, we are not in a position to make any definitive comment on the comparative success or 
otherwise of these institutes. 



2 TAPE institutes in Victoria have powers to hire and fire permanent staff. 

^ Three institutes in South Australia, two institutes in Victoria, two institutes in New South Wales, and two institutes in Queensland 
participated in the study. 

We define effectiveness to mean the achievement of objectives and targets, and efficiency as the achievement of objectives and targets 
within resources provided, or with minimum wastage. 
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Governance arrangements 

All TAPE institutes are public organisations under the purview of ministers responsible for 
education and training and the vocational education and training (VET) legislation for their 
particular state or territory. They negotiate annual funding for government business with officials 
from departments of education and training (or their equivalents). These plans and performance 
agreements with governments identify key performance measures (KPMs) and key performance 
indicators (KPIs)^ used to measure effectiveness and efficiency.'’ 

TAPE institutes in South Australia, Tasmania, New South Wales, the Australian Capital Territory 
and Western Australia operate under a centralised system of governance. In Queensland, two 
TAPE institutes have become statutory authorities directly answerable to the Minister of Education 
and Training, while the remainder operate under a centralised system reporting through the 
Department of Education, Training and the Arts (DETA). Statutory authorities and other TAPE 
institutes must negotiate funding arrangements for government business with government 
vocational education and training (VET) financial officers (VET investment officers in 
Queensland). TAPE institutes in Victoria operate either as separate entities with their own 
governing boards, or as TAPE divisions of dual-sector universities. 

In some states the corporate sharing of administrative functions means that system-wide 
management information systems operate alongside institute-specific ones. With the exception of 
the South Australian institutes, all TAPE institutes in this study prepare their own institute-specific 
annual reports. South Australian institutes provide information to be incorporated into the 
Department of Purther Education, Employment, Science and Technology (DPEEST) annual 
report. 

We did not set out to measure whether different governance arrangements result in better processes 
for evaluating effectiveness and efficiency. However, we find that institutes in both centralised and 
devolved systems used broadly similar strategies. We find that the ability of institutes to make 
decisions that make sense locally helps them to customise training to meet client needs. This 
happens in both centralised and devolved systems. 



Getting the plans right 

Planning is important to making good decisions and preventing the need for remedial action. 
Institutes have implemented various review mechanisms to ensure that their plans are right, but 
they acknowledge that even the best plans may not work, especially when actual demand exceeds 
projections. Nevertheless, the focus on thorough planning indicates a proactive approach to 
achieving targets. A retrospective approach is applied for evaluating course and module load 
completions, and student achievement and outcomes, including student and employer satisfaction 
with training. 



^ KPMs are meant to be generally few in number, strategic in intent, quantitative and have well-defined scope. They are intended to track 
performance over time. KPIs, on the other hand, can be either quantitative or qualitative and more ad hoc. These are often used to 
monitor the effectiveness of specific TAPE programs. 

^ In appendix B we provide examples of the types of KPMs and KPIs used by institutes to measure performance in reports to 
governments. 
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Setting objectives and targets 



Institutes have generally adopted the concepts of formal strategic planning. - In the main, a 
‘cascading approach’ is adopted. Typically state strategic plans will also have some overarching goals 
(see appendix A) and measurable objectives, which will translate into the strategic plans of the 
government departments responsible for VET and then be reflected in the goals and objectives. 
These flow into the work plans and budgets for teaching sections and other business units.® TAPE 
institutes in both centralised and devolved governance systems have considerable flexibility for 
developing their own institute-specific goals and objectives, which may be influenced by the 
program objectives of federal funding agencies and local needs analyses. 

Institute organisational goals that transcend or complement externally contracted targets for 
government funding generally relate to: 

A positioning the institute as a premier and quality provider of training 
A raising educational outcomes of specific groups and of students in general 
A achieving success in competitions and awards 
A remaining relevant to industry and the community 
A growing business from industry or international cHents 
A adopting innovative and quality teaching and learning practice 
A providing leadership in the community and fulfilling community obligations 
A being a desirable place to work and study. 

Negotiating the purchase agreement 

All institutes, regardless of their governance arrangements, negotiate a resource or purchase 
agreement with their respective state governments. This agreement, which is updated annually, 
details the number of ‘annual student hours’ that the state is prepared to purchase from TAPE and 
takes into account federal and state government funding priorities. Pinancial equivalents are then 
based on the price that the state will pay for a unit cost for training for particular programs. This 
price will reflect the cost variations that apply to different programs. Typically, state government 
VET investment officers’ (or their equivalents) make decisions about where to increase or reduce 
government training investments and start negotiation processes. 

In preparing a bid for funding TAPE institutes typically take into account their recent provision, the 
number of students applying for courses or continuing in programs and other known influences on 
training demand. Information on local job vacancies, projected commercial and industrial 
developments and other evidence of economic growth or downturn are gathered. Institutes also 
take into account any market intelligence from state education and training departments, local and 
regional economic development boards, local government databases. Commonwealth departments 
(especially about the international student market), and their own commissioned market research. 
Extensive networks with industry at director and teaching unit level are also used to understand the 
market for training. Purthermore, some institutes have employed a dedicated researcher to compile 
reports of published labour market information and economic projections. According to one senior 
manager, ‘The biggest risk is not lack of information, but pinpointing which information is relevant 



One institute has a ‘charter’ which performs broadly similar functions to the strategic plan. 

^ This has not always happened in this order and there have been instances where institute strategic plans have preceded the strategic 
plan of the government department. 

^ This term is used in the Queensland VET sector to identify government officials responsible for negotiating the purchase agreement. 
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and business-critical and which [information] can be ignored’ (Senior Manager, Quality and 
Business Improvement). 

When institutes receive the government’s purchase proposal, they commonly compare government 
decisions with their own bids or projections. Where there are discrepancies, institutes need to make 
a business case for increased funding. The negotiation with centralised VET agencies is a dynamic 
and interactive process which precedes the establishment of the agreement and the targets. This 
negotiation continues throughout the life of the agreement — should institutes require more funds — 
and in preparation for the next year’s bid. 

The need to develop a bid for the purchase agreement is the catalyst for TAPE institutes to apply 
considerable effort to strategic planning and to the development of information systems and 
processes for monitoring performance. Such processes can then be used to monitor the 
achievement of other internally set objectives and targets. 

Based on our conversations, we find that the answer to whether these processes are effective or 
efficient lies in the quality of the information that is used to arrive at a realistic funding bid and 
eventually a good purchase agreement. The ability of state departments and TAPE institutes to 
achieve a mutually beneficial funding arrangement depends on their access to good information and 
analyses that help them to make relatively accurate forecasts of immediate, intermediate and long- 
term demand. This continues to be a challenge and there is a need to invest more widely in activities 
to help institutes develop more realistic bids for funds. 

Understanding their markets 

Getting the plans right is predicated on institutes understanding the markets in which they operate 
and understanding what their competitors are doing. Institute-specific market research activities and 
those conducted by contracted market research organisations help institutes to understand what is 
happening in their local or broader environments. Typically they will also access information on 
competitor course offerings and durations from competitor websites or newspaper advertisements. 
They monitor costs of courses and fees either by undertaking ‘secret shopping’”’ activities, checking 
advertisements in local papers, or having individuals enrol in courses. 

Increased competition from all institutes within and between jurisdictions has encouraged institutes 
to enter non-traditional niche markets and to apply business models (including the franchise 
model”) to selling their products within and across state borders. ‘My markets are everywhere’, said 
one institute director whose franchise for small business training had been purchased by interstate 
providers. 

Having apprentices and trainees from different enterprises in the same classes also provides 
institutes with timely opportunities to gather market intelligence. In this environment, students 
discuss technological changes being applied at their workplaces and comment on events that have 
had an impact on delivery, including economic downturn or industry expansion. This valuable 
information can be used by educational program managers to identify areas of decline or of training 
demand, with a subsequent adjustment of course offerings to meet changed economic 
circumstances. 



Secret shopping is a tool used by market research companies and individual organisations to measure the quality of retail service or 
gather specific information about products and services. Secret shoppers posing as ordinary customers perform specific tasks, for 
example, purchasing items, rating customer service, asking questions, registering complaints etc., and then provide detailed reports or 
feedback about their experiences to their employers. When secret shopping is used in an educational context the institute is applying 
these techniques to find out about the courses being offered by other providers. 

The franchise model refers to businesses allowing other businesses to use the same concepts, designs and practices for a fee. 
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Nevertheless, the dearth of easily accessible and accurate data for making informed decisions about 
training demand concerns many directors and their managers. Institutes want readily accessible 
statistics on training demand, preferably located in a single source. TAPE institutes in New South 
Wales have access to a comprehensive document which details statistical predictions on the demand 
for each occupation group in each regional catchment area. This information aims to help institutes 
to understand general changes in their economic landscape. It is felt by some senior planners in 
institutes that such predictions, although of some interest, rarely provide reliable or robust estimates 
of demand that can be used at specific teaching program levels. This may require closer attention to 
local developments, which can only be provided by the institutes themselves. In other states the 
arrangements are more ad hoc. 

Educational managers and planning personnel believe that national statistics about sklU shortages 
are deficient for both metropolitan and regional areas. This is especially true for locations where the 
labour market is dominated by small and medium-sized enterprises. 

Institutes continue to be concerned about the lack of information on the volume and nature of 
private provision and the preferences of students. Such information would help them to understand 
the market for training and consequendy enable them to offer the types of programs they desire. 

In a more competitive environment institutes will have to become more adept at understanding 
their markets, especially in metropolitan areas where there is greater competition from other 
providers vying for students and government funding. One consequence of aU providers 
undertaking their own market research activities is considerable duplication of effort. This has the 
potential to reduce the amount of money allocated to the core business of teaching and learning. 

Institutes say they need to understand the extent of skill shortages. It is possible to gauge current 
levels of local demand using published material on job vacancies in local newspapers and national 
media. However, these may not provide good data on future training demand. Skills forecasting is 
notoriously difficult (Richardson 2006 )“. 



Understanding their government and commercial business 

It is clear that TAPE institutes are committed to understanding their business and that they need to 
remain so, if they are to remain viable. This means meeting purchase agreement targets and 
supplementing these with profits from commercial business. 



Customising the purchase agreement to local needs 

The purchase agreements signed with state governments are based on estimates of what institutes 
believe they will be able to deliver. It is only when the student enrolment period has ended that 
institutes have a better idea of the number of hours that may be delivered in the semester. Where 
annual student hours are not needed by one teaching area, ‘trading’ or a re-allocation of hours 
between teaching sections or faculties takes place. The first goal is to re-allocate within faculties and 
then to put any surplus hours up for re-aUocafion amongst other faculties. Recently, faculties 
responsible for the traditional trades have been the beneficiaries of a great number of these re- 
allocated hours. Hours are rarely re-aUocated between institutes; when this does happen it is done 
via the state training authority. 

This means that institutes are actively engaged in aligning their overall strategy to practical realities. 
This constant readjustment requires institutes to be especially dependant on good information 
systems. 



“ Richardson, S 2007, What is a skill shortage?’, NCVER, Adelaide. 
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Regular budget reviews 



Once budgets have been approved, a regular budget review (also reflected in performance reviews 
of managers) is the main strategy used to track progress towards financial and non-financial targets 
at work team, faculty and institute level. Typically, it is the role of educational managers to review 
budgets for their teaching sections, and faculty directors (or their equivalents) to review the 
financial progress of the faculty as a whole. In a number of institutes financial officers assist 
managers in faculty or business units to interpret emerging budgetary trends and identify potential 
issues early on in the budget cycle. Budgets are monitored against targets for revenue and 
expenditure, including costs related to staffing (for example, sick leave, workers’ compensation, and 
any other associated costs). Employee costs and supply and service costs as a percentage of total 
revenue are also examined. Statewide and institute-specific online financial systems, more 
sophisticated in some states and institutes than others, help institute directors and senior managers 
track progress against their overall budgets. 

When it is clear that programs are unlikely to meet their completion targets for the year, institutes 
may adjust delivery and assessment mechanisms to make up the shortfall. This includes increasing 
online delivery, clustering competencies, or combining classes so that one teacher is allocated to 
larger groups of students with support from a tutor. It may also include increasing RPL (recognition 
of prior learning) assessments and supplementing courses with fuU-fee-paying students. Institutes 
also reduce the nominal hours suggested for a program. This accelerates the completion of 
programs to meet targets. However, this practice is rarely used for access and equity programs as it 
is felt that these students require all the hours allocated to a program. Such activities are clearly 
focused on efficient use of resources. 

Decline in the health of an industry will affect the viability of programs, and faculty directors and 
their educational managers need to be aware of any changes. For instance, the decline in demand 
from a specific sector of an industry may require a TAPE institute « to review the focus of the 
training provision. ‘You need to have as much planned as possible, and be on top of the plan. You 
also need a plan B if plan A fails and you need to change direction’ (Faculty Director, Trades). The 
impact of the global financial crisis may also require institutes to change direction to ensure that 
they are delivering skills training that will help students once the financial crisis has abated. 

The purchase agreement is the catalyst for institutes to establish mechanisms to measure progress 
towards targets. However, it is the commitment and energy of institute directors and their executive 
and managerial teams that ensure that attention is constantly focused on the business. The words of 
one institute director reflected a common sentiment, ‘We make sure we deliver what we have been 
contracted to or promised to deliver.’ 

In appendix B we give examples of the mandatory information that institutes need to provide to 
state funding bodies. 



Expanding commercial activity 

Institutes are acutely aware that they will need to grow their commercial business if they are going 
to survive in a more competitive environment. They are increasingly applying common business 
strategies to guarantee they meet or exceed their financial and non-financial targets. These include 
incorporating in-built margins for profits and for student attrition into costing structures and 
funding bids. 



One case in point was the sudden decline of demand for boat-building at the high end of the market (industr}'^ may be scaled back 
with some production being moved to China). This will affect the direction of apprenticeship training, especially for marine 
mechanics. Adjusting to the demand may mean shifting attention to skills required at the lower end of the market. Currently the 
institute is meeting with the industry and state department to determine the direction. 
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Institutes are keen to increase their fee-for-service programs to increase profits. They have ventured 
into new markets, within and across state borders and internationally, and have used profits to free 
up monies for programs considered to be high priority. In some institutes high demand for 
apprenticeship training takes precedence over fee-for service courses aimed at up-skiUing or 
licensing existing workers (including trades). This means that fee-for-service courses may only be 
run at certain times of the year when they can guarantee that the course will be full. Classes for such 
programs are advertised well in advance so that if the course does not fill it can be postponed to a 
later time. 

Additional revenues can be derived from state and federal government special initiatives, alternative 
sources of students (including existing workers and international students) and new industry and 
community partnerships. Institutes often cross-subsidise programs that may be at risk of not 
meeting targets with funds derived from commercial programs. These funds are also used to fund 
internal review and improvement processes. The development of measures which bring together 
cost inputs and employment outcomes is an extension of these. The pooling of profits produced by 
different commercial programs to finance strategic reviews and strategic projects helps institutes to 
stay focused on effectiveness and efficiency through continuous improvement. 

Institutes are keen to enter national markets for training, but cross-border regulatory anomalies are 
seen as major barriers to establishing training provision for national enterprises. Accurate 
knowledge of the international market is also difficult to come by. Institutes are often dependent on 
their relationships with education agents in key countries and the information provided by the 
government agency, Australian Education International. 



Benchmarking their performance against other organisations 

Institute directors and their managers have embraced the concept of evidence-based evaluations for 
determining effectiveness and efficiency. Some institutes have built local online systems and 
processes which enable them to have immediate access to running summaries of training and 
budget activity. There is a keen focus on the ‘bottom Hne’, although the committed approach of one 
institute director who claimed ‘If we can’t measure it we don’t do it’ is not always echoed by others. 

Ready access to state online databases or reports enables Queensland institutes to compare their 
own performance with state averages and with other institutes. This enables them to monitor how 
they are progressing against financial and non-financial targets and against state averages. This 
information prompts institutes to find explanations for their own over- or under-performance and 
to implement remedial action. 

The results of a range of external surveys like the national NCVER Student Outcomes Survey 
(SOS), the Survey of Employer Use and Views (SEUV) and state-based surveys of business 
satisfaction (where available) are used both for strategic planning and evaluation. Findings help 
managers to track trends in client satisfaction and employment outcomes and to understand areas 
of improvement and decline. The Student Outcomes Survey is often used by institute marketing 
departments. Conducting external audits for other institutes is another benchmarking mechanism. 
One issue that relates to the effectiveness of these practices is that, in many instances, survey 
response rates and sample sizes are too low to provide any meaningful information at program 
level. Another issue relates to the currency of the information, when analyses are only available once 
students responding to the survey have left the organisation. 

NCVER publications on students and courses and apprentices and trainees are consulted more 
frequently by directors and senior administrators than by education managers or teachers. The 
NCVER publications help institutes get a general view about how they are travelling in terms of 
participation. 

Surveys of staff conducted by state agencies enable institutes to monitor staff numbers, 
characteristics, turnover and absenteeism. In some institutes these surveys, along with institute- 
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specific surveys of staff (including exit surveys), are used as an indication of the health and culture 
of the organisations and in the identification of areas for improvement. In others a staff ‘intentions 
survey’ enables institutes to develop organisational development responses for succession planning 
and for mentoring and coaching junior staff. There is some use of exit interviews with staff and 
students but these are not widespread. 

Monitoring costs for training and services 

AH jurisdictions have determined a unit price per annual student hour” which reflects cost 
variations between different programs, including costs of management and teaching and 
administrative staff and costs of goods and services. Such an indicator is often used by states and by 
institutes for benchmarking performance against other providers. It may also be used to work out 
whether programs are profitable and whether any major restructuring must occur. One institute has 
come up with a measure for evaluating the continued viability of different programs by integrating 
targets, staffing and costs. This has also been used to launch and implement workforce cross- 
skiUing, upsizing, or downsizing. Although other institutes may not have such a sophisticated 
method for understanding their cost-efficiency in real terms, they are also interested in calculating 
unit training costs to re-evaluate funding and staffing. 

Most institutes are aware that their permanent staffing costs are the major component of their 
expenditures and, although they may be able to reduce numbers of casual staff, it is more difficult 
for them to cut the numbers of permanent staff. This is especially true where states have 
implemented policies to protect employment of public servants. Institutes have no control over 
state-negotiated industrial relations settlements, which influence costs for staffing. This is the case, 
regardless of governance arrangements. 

Nevertheless, institutes commonly try to understand whether their staffing is adequate. One way 
this is done is to calculate a ratio of teacher contact hours to full-time equivalent staff If there is a 
mismatch — more staff than required for a certain area (who cannot be justified by special 
circumstances) — there will be a need to work out ways for ensuring that staff are relocated to areas 
where required. If they are still surplus to requirements, there are generally processes in place to 
offer them targeted separation packages. Victorian institutes have more autonomy to ‘hire and fire’ 
staff. 

Expecting all institutes in a jurisdiction to apply a uniform cost per hour in a particular program is 
perceived to be an issue by some managers. This is because the cost does not take into account the 
‘real costs’ of delivery in some rural and remote locations and for programs (including flexible 
learning programs) in which students require considerable personalised learning support. 

There is a perception that each year the budgeting process becomes more difficult because of state 
cost-saving measures. Institute directors and their senior managers accept the need to establish 
realistic and routine cost savings. 

Several institutes have their own online systems for helping faculties track their finances and costs 
for training. The lack of fully integrated statewide financial and human resources systems continues 
to limit the ability of institutes to derive an exact picture of training costs. ‘If we want to know how 
much it costs to train trainees we can’t make an accurate costing; it is only an estimate [albeit] a 
good estimate,’ commented one finance manager. 

Not everyone thinks that benchmarking oneself against other providers is useful. A small group of 
TAPE leaders is of the view that it is much more fruitful for TAPE institutes to benchmark 
themselves against their own prior performance. The notion of implementing a ‘fair price’ for 



Queensland and South Australia use the term ‘Annual hours curriculum’ (AHC), New South Wales uses Annual Student Hours 
(ASH) and Victoria uses Scheduled Hours. 
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annual student hours was also mentioned, with ‘fair’ meaning that institutes would not be 
disadvantaged by having to meet any unanticipated occurrences like the salary costs of staff who 
have extended sick leave, along with the cost of recruiting and paying for substitute staff. 

Institutes want national statistics to provide meaningful and reliable data about total VET provision, 
as well as data they can use to benchmark to their own provision at the course level. AH institutes 
submit data to state agencies as part of their national and state reporting responsibilities. 
Mechanisms to ensure that institutes are able to validate their own information may improve 
accuracy and streamline reporting. 

Comparisons of delivery costs across jurisdictions are also perceived to be problematic. This is 
especially the case in jurisdictions whose benchmarked costs turn out to be higher than the national 
average. These concerns could be addressed by ensuring that attempts at benchmarking costs use 
the same items in the mix to calculate unit costs. This includes being aware that different award 
conditions regarding maximum work loads for full-time teachers may also affect the cost of 
delivery. 

Sharing information with others 

It is common for institutes within and between jurisdictions to share non-commerciaUy sensitive 
information about teaching and learning, management practice and financial reporting. This is 
regardless of governance arrangements. Teaching and administrative staff are supported to attend 
regular meetings of program or function-specific groups and networks. State agencies and networks 
for curriculum development and maintenance (or their equivalents) continue to play a critical and 
welcome role in providing access to information on policy initiatives, new legislation, and the 
moderation and validation of assessments. Institute directors and senior managers also visit other 
institutes across the country to learn about new programs, systems and processes. 



Meeting client needs 

We divide the major clients into industry and students. 



Meeting industry needs 

Institutes commonly employ similar formal and informal mechanisms for gathering intelligence on 
what industry wants. Formal course advisory panels, industry forums and other industry 
engagement arrangements help them to keep up to date with workplace change, advances in 
technology and industry requirements. The employment of part-time instructors from industry also 
helps them to maintain relevance and currency. 

AH institute directors and many senior managers sit on key state, regional or national committees 
and boards. Conversely, many influential industry leaders sit on TAPE boards and councHs. These 
high-level forums are especiaHy usefiH for gathering inteHigence on emerging issues and developing 
relationships with key employers. This is another indication of their focus on proactive planning 
activities to ensure they are able to respond to industry needs. 

The majority of educational managers and many teaching staff (especially in the trades) are 
members of industry networks and associations. Through these networks, institutes aim to increase 
their understanding of commercial and industrial developments. This helps signal the emerging skiH 
requirements and assists in identifying employers who need training for workers. There is scope for 
them to market new courses or ‘up-size’ existing training provision. In addition, teacher— employer 
collaboration in the development of training plans, teacher or assessor visits to students in 
workplaces, consultations with employers and assessment of workplace training also enable TAPE 
institutes to understand whether students are performing to industry standards. Institutes gather 
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feedback on how well they are preparing students for the workplace through close partnerships 
with industry in delivering on-the-job training for apprentices and trainees and providing work 
experience and placements for students in non-trade programs (especially for nursing). 

Institute directors and their senior managers use the results of a range of external surveys, including 
state-based surveys of business satisfaction (where available), for both strategic planning and 
evaluation. Findings help managers to track trends in client satisfaction and employment outcomes 
and to understand areas of improvement and decline. National surveys like the NCVER’s Survey of 
Employer Use and Views are used by senior managers. The recently released national employer 
satisfaction surveys are aimed at helping institutes to collect the type of information required for 
compliance with AQTF 2007 audits. As such surveys have only recently been introduced, it is still 
too early to tell whether they will provide the information required. 

A Customer Relationship Management system being purchased and trialled by some institutes 
allows them to track interactions with customers and identify their most important clients. Similar 
commercial systems are being trialled nationally. Project planning, monitoring and evaluation 
through traditional project management systems complement these new approaches. 

Other strategies which are proactive attempts to attain good outcomes for industry and for 
apprentices are industry-specific programs that combine mentorship and training for apprentices, 
for example, in commercial cookery courses such as the ‘Appetite for Success’*'’ program. This is 
aimed at improving training completions and retention in the industry for apprentice chefs. 

Despite these broad industry-engagement practices, institutes claim that it is not always easy to 
understand what employers want. Substantial groups of employers do not participate in advisory, 
collaborative or information-sharing activities and their expectations often vary and can be 
contradictory. For example, some institutes have found that employers may want the majority of 
apprenticeship training to be done in the workplace, while others want all the training to be done at 
TAFE. Although institutes say they do their best to adapt provision to meet individual needs, the 
question of ‘who speaks for industry’ continues to be an issue. 

TAFE institutes collect extensive information about training needs in their local areas, but gaps 
remain. Institutes want ready access to accurate predictions of skill shortages and training demand. 
This gap might be addressed by planning departments of state training authorities or by regional 
economic development boards, where they exist. It could also be a task for state-based training 
advisory bodies. The forecasting of skill shortages is not an easy task, no matter who takes 
responsibility for it. 

Institutes require more information about the characteristics of businesses (especially small 
business) in their catchment areas and the training needs and values of enterprise clients. They also 
want further information regarding employer expectations of training and evaluations of whether 
the training for new and existing workers has been relevant, useful and successful. 



Meeting student needs 

Institutes are focused on maintaining current and accurate records of outcomes, especially as these 
are required for funding purposes. Key outcomes for the institute as a whole and for individual 
programs include enrolment numbers, subject completions, module load completion rates, course 
completions, retention rates and attrition rates (for specified groups identified in the purchase 



Under AQTF 2007, registered training organisations are required to collect and use data on three quality indicators endorsed by the 
National Quality Council (NQC). These are: learner engagement, employer satisfaction, and competency completion. 



The ‘Appetite for Success’ program aims to provide support to apprentices throughout and beyond their apprenticeship to improve 
retention of chefs in the trade. 
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agreement). Outcomes may include the percentage of students who gain employment in any job or 
in their specified field. Institute performance in national and international competitions and awards 
are also used. These methods are also invaluable for evaluating the institute’s performance in 
meeting student needs. 

Common strategies for giving students ‘a voice’ include focus discussion groups, one-on-one 
discussions with teachers, representation on governance bodies, and complaints and suggestion box 
arrangements. Student responses to internal course or program surveys (run at teaching section 
levels by teachers or by institute quality units) provide information on issues related to teaching and 
learning, resourcing and areas that need improvement. 

Institutes say they are poor at tracking what happens to students once they have completed courses 
and left the institution. There are two questions of interest to institutes: whether students are able to 
move into better jobs; and whether they progress to higher-level qualifications.” One institute 
director commented, ‘We need a better fix on the destinations of students, [including] those who 
leave before they complete’, and this observation was echoed by other senior managers. There are 
also concerns about the difficulty of getting accurate information on training contract completions 
and data on those who do not choose to come to TAPE at aU. The lack of good alumni strategies is 
felt to inhibit the ability of institutes to conduct longitudinal surveys or to develop samples of 
respondents. Institutes that have implemented exit surveys continue to be frustrated by low 
response rates. 

As with domestic students, a formal process for monitoring the destinations of international 
students would also help institutes to assess the success of their programs. Institutes claim that they 
develop strong relationships with international students while they are in training, but the lack of 
formal alumni strategies means that institutes lose touch with the students once they have 
completed their studies. Another set of problems faced by many of the TAPE institutes often 
relates to the low levels of English language skills that international students come with when they 
enrol in TAPE programs. Teachers report that, while agents responsible for promoting 
international programs are focused on attracting as many students as possible, teachers are 
concerned about the ability of these students to complete the training. 

Institutes are keen to understand the effectiveness of their training and the extent to which it has 
been successful in the workplace. A number of respondents suggest the need to complement 
destination surveys with one-on-one conversations and focus groups with TAPE graduates and 
employers in workplaces. Keeping in touch with students is felt to help institutes in other ways, as 
institutes could contact students who leave without a qualification and encourage them to complete 
their qualifications through recognition of prior learning. This is especially important if 
Commonwealth apprenticeship and traineeship incentives continue to be attached to completion. 
Quality units (or their equivalents) within institutes are already responsible for conducting internal 
surveys of students. There is no reason why such surveys could not also be conducted by these 
units. 

Low rates of survey responses to internal and external surveys reduce the reliability and the 
usefulness of findings for institutes. Institutes report that students comment mostly about 
personalities of teachers and sometimes the adequacy of services, rather than the quality of course 
content. It is also felt that the value of survey findings at program level is limited because analyses 
are only completed once the cohorts of respondents to these surveys have left the institute. 
Information, which aims to improve the training experience of students therefore cannot help these 
students. Offering incentives for survey participation and providing students with ready access to 



The Student Outcomes Survey, conducted by NCVER, provides this information to some extent and is used at a high level in 
strategic planning. Institutes want more institute-specific data to help them better evaluate course-specific outcomes. 
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online feedback processes have been identified by respondents as possible ways to improve survey 
response rates. 

There is a perception that the push for increasing trade training, evident in the last few years, has 
resulted in fewer annual student hours for access and equity programs. There is also a concern that 
TAPE access and equity programs are used as alternatives to school for adolescents (generally 1 5- 
year-olds) at risk of becoming disengaged. Lack of prior knowledge about the poor standards of 
literacy and numeracy and the behaviour of these students is felt to hinder the ability of TAPE 
teachers to prepare sufficient resources to help students overcome these difficulties. 



Applying quality assurance measures 

Another indicator of effectiveness and efficiency of programs and practices for all TAPE institutes 
is their performance in AQTP 2007 audits, as well as in audits performed by other external quality 
assurance organisations. “ Quality units generally support the need for quality principles to be 
integrated into the daily operations. Compliance results are used to indicate areas for improvement. 

Institutes are especially aware of the need to implement systematic assessment, moderation and 
validation processes and to have adequate industry engagement in these processes. At the time of 
writing, institutes were awaiting the results of the national surveys that have been created to 
evaluate student and employer satisfaction with training. It is anticipated that these will help them to 
provide the types of evidence of achievement that meet standards for outcomes-based audits. How 
institutes are to measure successful ‘outcomes’ continues to raise debate. 

As well as national quality assurance standards, institutes must satisfy other legislative requirements 
(including licensing and occupational health and safety) and standards of industry course 
accreditation bodies. They must also meet the standards of international bodies, where required (for 
example, Lloyds of London maritime standards). In a number of institutes the review of processes 
and practices is an important method for addressing issues arising from quality audits and the 
implementation of continuous improvement. Quality assurance processes are dependent on the 
commitment of all staff and the maintenance of adequate documentation and records. Quality units 
have commented that some staff members are more committed than others to upholding quality 
standards. An ongoing concern for many quality team staff is ensuring that quality principles are 
integrated into daily workplace practice. Another concern is the low level of response to internal 
student surveys. 

International programs 

Institutes with international students have to meet a number of additional compliance regimes. 
Institutes must meet extra legal requirements associated with Education Services for Overseas 
Students (ESOS) and the Commonwealth Register of Institutions and Courses for Overseas 
Students (CRICOS) standards. These are the key standards for managing parficipation, attendance 
and performance of international students and ensuring the quality of provision. The progress of 
international students (especially those who may be at risk of failing) is a key element of these 
standards and requires educational managers to be especially vigilant. 



The concepts and strategies of the Business Excellence Framework have been adopted by a number of institutes. These relate to the 
provision of effective leadership through clear direction and goal focus; using information on what customers and markets need and 
value to drive organisational practice, products and services; applying concepts of continuous improvement, systems thinking and 
innovation; using data, information and knowledge to improve strategic and operational decision-making; behaving in ethical, and 
socially and environmentally responsible ways; and focusing on results, values and outcomes that are sustainable. 
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Institutes are concerned that standards for entry into TAPE programs in Australia for international 
students are not rigorous enough. For example, students require an International English Language 
Testing System (lELTS) score of 5.5 for admission to TAPE programs; yet this standard does not 
ensure that students will be able to perform the requisite tasks. 

Ensuring the quality of offshore training provision is also a concern. TAPE institutes need to 
ensure that their offshore partner organisations guarantee that trainers have the necessary training 
and assessment qualifications and use the assessments that have been agreed in learning plans. 
Guaranteeing that the requirements are being adhered to is not always easy. 



Conclusions 

The need to evaluate effectiveness and efficiency in TAPE institutes has generally been driven by 
accountability for government funding and compliance with legislative and quality assurance 
regimes. These obligations have prompted the establishment of processes for monitoring 
performance against government financial and non-flnancial targets. They have also acted as 
stimulants for developing similar processes for tracking performance against other institute 
objectives. This holds for TAPE institutes that operate in either centralised or devolved governance 
arrangements. There continue to be critical gaps in the information that TAPE institutes require to 
better understand their markets and their performance. These gaps include accurate forecasting of 
skin demands and more comprehensive information on employer needs and student destinations. 

We have also found that TAPE institutes want to link outcomes to specific training interventions 
and management practices. They are highly committed to being proactive about planning and 
monitoring their processes to ensure that they achieve their targets. Although evaluating past 
performance is valuable, there is a sense that adequate planning prior to the commencement of a 
program and constant review and remediation throughout the life of a program are far more 
important strategies for ensuring that targets are met. 

How are TAPE institutes placed to operate in an environment which is more competitive and less 
secure? TAPE institutes have developed considerable experience in increasing their business in 
commercial markets and have been doing so since the early 1990s, when the regulated training 
market was opened up to the private sector. They have also learned the importance of 
understanding their markets. These experiences should help them in a more competitive 
environment. 

It appears that governance arrangements have little bearing on the ability of institutes to monitor 
their effectiveness and efficiency. We must be mindful that our study has only looked at how 
institutes operate in the current environment, where funding arrangements are centrally driven. We 
cannot say how governance arrangements might affect institutes if such funding arrangements were 
to be devolved. 
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Appendix A 

State strategic objectives 



Table 1 Strategic objectives for South Australia, Victoria, New South Wales and Queensland 



South Australia Victoria New South Wales Queensland 



1 Ensuring that South 
Australians have the 
necessary education 
and skills to participate 
in the high-skills 
economy 



2 Providing high-quality 
employment and 
workforce 

development services 



3 Ensuring that young 
people are supported 
in reaching their full 
potential and actively 
engaged in learning 
training, work and in 
their communities 

4 Providing a 
coordinated whole-of- 
government approach 
to the development of 
an innovative 
community 

5 Building a high- 
performance 
organisation 



1 To achieve a Victorian 
workforce with the mix 
and level of skills and 
knowledge required to 
meet the current and 
emerging needs of 
employers, industry an 
the community 

2 To achieve increased 
workforce participation 
through enhanced 
access to skilling 
opportunities, 
especially for groups 
with currently low 
participation rates 
(unemployed and 
under-employed) 

3 To achieve equitable 
access to skills 
development 
opportunities for all 
individuals to develop 
the full range of their 
potential skills and 
employability 

4 To achieve provision 
of systemic 
opportunities for 
existing workers to 
build towards higher 
qualifications 



1 Ensuring improved 
social and economic 
wellbeing for the 
people of NSW 
through the provision 
of high-quality 
education and training 

2 Delivering better 
services by: 
increasing levels of 
attainment for all 
students, and 
ensuring that more 
students complete 
Year 12 or recognised 
vocational training 

3 Growing prosperity 
across New South 
Wales: with more 
people participating in 
education and training 
throughout their lives 



4 Provision of better 
access to training in 
rural and regional 
NSW to support local 
economies 



1 Every child and young 
person will be well 
prepared for life 
success through 
learning and education 



2 Queenslanders will be 
linked to opportunities 
that support economic 
prosperity and 
enhance lifelong 
wellbeing 



3 Queenslanders will 
benefit from a rich 
cultural life and thriving 
creative communities 



4 Productive 

relationships will be 
fostered to maximise 
outcomes for 
Queensland 



5 Qur people’s talents 
will create a dynamic 
and innovative 
organisation 
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Appendix B 

Examples of information that states are 
required to provide to state funding bodies 



This information refers to 2008 requirements. 



South Australia 

Institutes must provide their state training authorities with information on student, financial and 
workforce activity, which is then used to develop a set of productivity measures. 

Student activity indicators comprise: 

A number of students, subject enrolments, annual hours curriculum, module load completions 
A percentage of module load completion to annual hours curriculum. 

Hours are reported against state and Commonwealth Government recurrent and special purpose 
funding (including FSI 500” hours in the Resource Agreement, Aboriginal education. User Choice, 
SA Works program, other special funding, and fee-for-service, including commercial ventures and 
overseas students and overseas delivery activity). 

Financial indicators are reported in terms of: 

A expenditure on teaching and non-teaching salaries, goods and services, and capital 

A total revenue from non-government-funded general revenue, student fees and fee-for-service, 
and government-funded general revenue. User Choice and recurrent allocations 

A operating result (one set of figures based on the inclusion of capital expenditure and another 
based on its exclusion). 

Workforce indicators are reported in terms of effective full-time equivalent staff. They include 
teachers (including lecturers and hourly paid instructors) and non- teaching staff). 

Productivity measures are reported in costs per curriculum hour. These are calculated in terms of: 
A curriculum hours (including gross total expenditure for all FSI programs) 

A net cost to the state based on total expenditure less non-government-funded revenue 
A total expenditure less the total revenue (operating result). 

It is in the reporting of these mandatory requirements that institutes need to keep in mind the push 
to keep the unit cost data down. Institutes are particularly keen to ensure that the real costs of 
delivery in certain locations are measured. This is especially so for Regional Institute which has a 
much larger geographical footprint across the state, than the other two institutes. As such it is 
particularly prone to downturns in regional industries and or labour markets. 



” Fund Source Indicator 500 refers to funding allocation under state recurrent funding for courses. 
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of regional industries and/ or labour markets. In the case of industries collapsing in towns or 
regions, the institute will have to modify its targets to take into account changed circumstances. 



Victoria 

Victorian institutes also need to provide Skills Victoria with information on how they have fared 
against the purchase agreement, including their performance on ASH targets for P (profile funding), 
L (apprenticeships and traineeships), Q (pre-apprenticeships and advanced trades) and LQ (special 
projects). Institutes also report on staffing, revenue for FTE and increase in growth of fee-for- 
service programs. The VET in Schools programs auspiced by TAPE institutes or delivered as 
stand-alone programs are provided as fee-for-service programs. These mandatory reporting 
requirements measure what institutes have been contracted to deliver. 



New South Wales 

Institutes in New South Wales also report against objectives of state— Commonwealth bilateral 
agreements, state programs, and institute purchase agreements. In aU cases the major indicators are 
annual student hours (ASH), enrolments, student numbers and completions. 

Commonwealth Government Programs; Skilling Australia’s Workforce 
Initiative (SAW) 

A total ASH (divided into SAW core and fee-for-service ASH) 

A SAW enrolments at certificate III or above for Indigenous students 

A SAW enrolments for disability students 

A SAW enrolments for 20 to 24 and 45 to 64-year-olds 

A SAW commencements in trade and non-trade skill shortage areas 

A SAW total enrolments for Indigenous students 

A non-core, core and total SAW recognition of prior learning (RPL) 

A SAW fee-for-service RPL. 

MCVETE commitments 

A completion of diplomas and above 

A average module completion rates for Aboriginal students 
A graduate satisfaction with overall quality of TAPE. 

TAPE NSW strategies 

TAPE Peak Education Group: Workplace delivery 

A increase in module/unit enrolments workplace/ employment-based delivery 
A increase in percentage of workplace of workplace/ employment based delivery (ASH). 

Best practice adjusted 

A progress against adjusted best practice (ASH generated through use of core funding). 
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Delivery of services 

A gross and net revenue from external activities 
A percentage of net revenue compared with all revenue 
A percentage of net commercial return 
A overall indicative cost per ASH 
A invalid enrolment rate. 

Queensland 

Institutes in Queensland (including statutory authorities) must report on specific performance 
according to resource agreements and state initiatives: These include: 

A ACH (annual curriculum hours) 

A number of high-level qualifications 

A percentage of RPL 

A number of 15 to 17-year-olds (Education Training Reform for the Future Initiative) 

A number of Indigenous students 
A number of mature-age students over 45 years of age 
A number of module completions 
A number of FTE students 
A number of apprentices and trainees 
A number of people with a disability. 
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Support document details 



Additional information relating to this research is available in Tracking our success: How TAPE 
institutes evaluate their effectiveness and efficiency — Case studies, which can be accessed from NCVER’s 
website <http:/ /www.ncver.edu.au/publications/2169.html>. 

The case studies 

ff Hunter Institute 

A North Coast Institute of TAPE 

Gold Coast Institute of TAPE 

Metropolitan South Institute of TAPE 

RMIT University 

Sunraysia Institute of TAPE 

Institutes in South Australia 

♦ TAPE SA Adelaide North 

♦ TAPE SA Regional 

♦ TAPE SA South 

A References 
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